Supporting Irregular and Dynamic Computations in Data Parallel Languages

Data-parallel languages support a single instruction ow; the parallelism is expressed at the instruction level. Actually, data-parallel languages have chosen arrays to support the parallelism. This regular data structure allows a natural development of regular parallel algorithms. The implementation of irregular algorithms necessitates a programming eeort to project the irregular data structures onto regular structures. In this article we present the diierent techniques used to manage the irregularity in data-parallel languages. Each of them will be illustrated with standard or experimental data-parallel language constructions. 1 Irregularity and data-parallelism First observe that data-parallelism and task parallelism programming models are derived directly from SIMD and MIMD execution models. The rst trace of data parallelism is seen in the rst supercomputers such as the Cray 1 or the Cyber 205 that provided a pipelined parallel execution model. The access to contiguous or regularly sparse memory slices supported matrix and vector algorithms that are very CPU time intensive. The following generations of pipelined machines possessed instructions to access sparse vectors in the memory (operations of \gather/scatter"). These new instructions introduced irregular data structure handling. Many applications using these type of structures (sparse matrix, Mont e Carlo...) then beneeted from the power of these supercomputers while preserving the vector programming model. With the emergence of parallel and massively parallel machines, where the memory is physically distributed on a large number of processors, new parallel programming techniques appeared. In order to preserve the knowledge base, a programming model similar to the vector model was proposed: the data-parallel model. Here vectors or matrices are distributed across all the processors. Parallel operations are processed simultaneously by the processors. Here again, regular structure handling involving only regular communication diagrams between processors have rapidly ben-eeted from the growing power of these new machines. On the other hand the use of irregular structures leads to irregular general communications; their implementation requires a particular eeort of the programmer that ensures a balance between processing and communication time by a correct distribution of data.